People buying or selling silver or gold outside this process would simply do so at the spot price. This system worked pretty well, and couldn't easily be manipulated, since real silver and gold was (presumably) changing hands.
However, most silver today is bought or sold based on up-to-the-minute spot prices, which are based on the futures market and the OTC (Over the Counter) market, both of which trade massive amounts of silver and gold (the sheer volume implies that it is for speculation, to take advantage of short-term price fluctuations; most of the silver or gold is never in the hands of the buyers or sellers). In the case of futures markets, computer generated trades appear to dominate, which essentially day trade, likely buying and selling within minutes.
In the United States, the COMEX is where the futures contracts are traded. Amazingly, about 0.00 ounces of silver are traded on COMEX for every ounce that is delivered!
Given how much silver is bought or sold by computer programs, and how little is delivered, it appears that the spot price is primarily set by the few people that do this massive amount of day trading in silver. Some people question whether or not the price of silver (or gold) is manipulated, and it is easy to see why. It almost certainly is manipulated, it's just a question of whether the people manipulating it are doing so intentionally or not.
First, the largest source of new silver by far is the mining companies. They have to sell their silver. In theory, they could store it waiting for a better price, but then they won't have money coming in to pay for their operations. Otherwise, they have to accept whatever the current price is. And people aren't going to buy for more than the spot price today (unless there isn't anywhere else to get it, but there almost always is).
The other major source of silver is recycled silver. The companies melting and refining the old silver don't care what the price is, since the silver wasn't theirs to begin with (they have to buy the silver that is being refined). Since they have to both buy and sell, they have to use the current spot price (again, since buyers won't pay more than the spot price).
For so-called 'retail silver' (such as silver eagles, smaller silver bars, and 90% junk silver coins), some of that comes to market through estates. The people selling off the estates aren't likely to know (or even care) whether the value of silver is higher than the spot price. They will just sell the silver for something close to the spot price. For those that have silver and are looking to sell it themselves, dealers aren't going to pay higher than the spot price (plus/minus a small premium depending on the item), so they need to either sell based on the current spot price, or hold on for a higher price later (but, the spot price could go down by the time they finally get tired of waiting).
So for the most part, the only people that might like to sell silver, but aren't willing to sell based on the current spot price, are those that are both convinced that the value of silver is much higher than the current spot price, and can wait to sell. And that accounts for a very, very small percentage of the market.
So if the market is manipulated, and the price is artificially low, that doesn't mean that silver won't come to the market.