In 1772, after the building of the Mission San Luis De Tolosa, events started to gain momentum. Ranchers and merchants came in, and as the population grew, so did the need for supplies. Shipping became common in the first half of the 19th Century using Cave Landing as the port 'o' call. But because passengers had to climb a rope ladder up and over the rocky cliffs and supplies had to be lifted up with massive cranes, Cave Landing's history was a short one.

Port San Luis came into its own when the wharves started to be built. There were stage lines and small gauge railroads created to run supplies to and from the Port and San Luis Obispo. Oil companies built pipe lines to the Port to ship out to all points in the Pacific, including shipping oil to Alaska. Finally, right after the Great Fire of San Francisco in 1906 ( and probably because of it), San Luis Bay was designated as an official United States Port of Entry. A $500,000 breakwater was built using rock blasted from Morro Rock and shipping from all over the world kept it quite busy.

The Port was under the jurisdiction of San Francisco and later Los Angeles. The deputy collectors would drive in whenever their services were required and had to use rented offices. An official building was needed, so David Muzio of Avila Beach, built what was The "Old" Custom House. The building was dedicated on March 19, 1927 and the largest celebration the County had ever seen was staged. Over 200 public officials and businessmen of California attended as well as a crowd of over 3500 on-lookers. There was a parade of over 200 decorated cars, and floats with a free lunch for everyone and "the Atascadero-Paso Robles High Schools Band in uniforms of maroon color" played for all.

Construction on the "new" Custom House restaurant started at the beginning of 2001. After 17 months of hard work, "The Custom House" Restaurant was finally able to open its new doors in July of 2002. The Custom House today specializes in a unique indoor/outdoor dining experience. Our menu boasts many old favorites as well as some new specialties. Our food is prepared fresh daily and we offer a large variety of exotic drinks to enhance your every meal. So even though it took over 385 years, old Juan Cabrillo would have been proud to see what has become of Todos Santos Bay.

Ed and Meme Fischer came from a long line of Fillmore citrus farmers. Like most people Avila Beach they fell in love with its beauty, purchasing a vacation home at the beach. Meme quickly became a neighborhood hit with her baking and was known for her cookies, nut bread, Italian dishes and most of all her buttermilk pancakes. What better way to celebrate her love for Avila beach then to offer her original recipe for buttermilk pancakes to all visitors, just as she did for decades earlier.