Came across this error while trying to setup a new search. The search did not return any results and the Event log for the WFE had the following error on it (note this was running SP Foundation, but the error could easily occur in any version).
Log Name: Application
Source: Microsoft-SharePoint Products-SharePoint Foundation Search
Date: 8/20/2012 10:47:20 AM
Event ID: 57
Task Category: Search service
A database error occurred. Source: Microsoft OLE DB Provider for SQL Server Code: 14 occurred 17 time(s) Description: [DBNETLIB][ConnectionOpen (Invalid Instance()).]Invalid connection.
This one was a bit tricky to track down. The solution was actually really simple and because I didn’t have full knowledge of the environment I was unable to track down the problem right away.
It turns out the database server I was trying to connect to did not make use of the default instance. Instead it had a custom instance setup. I didn’t initially notice this because since it was the only instance on the SQL Server I was automatically logged into it within SQL Management Studio even though I didn’t indicate I wanted to enter the default instance.
So the solution was to simply to ensure the search and crawl were pointed at the custom instance (they were pointed at the default). Once this was completed the jobs run, the search returned data. Just goes to show you how important knowing your environment is.