Shorten time to wait for element visible issue

Doing research to set an explicit wait time allotted to finding an elements visibility, it seems like this is the way to do it:

new WebDriverWait(driver, time).until(ExpectedConditions.visibilityOf(element));

When I give this a try, and set the wait time as short as 3 seconds, it seems like the script is still taking 30 seconds or more before it decides an element is not there. What am I doing wrong?