Generally, a small business is not required to have a CPA or certified public accountant. A CPA would be needed if the small business must have its financial statements audited or reviewed in order to obtain a bank loan, to apply for a grant, to bid on a job, or some other unique requirement.
Although most small businesses are not required to have a CPA involved, a small business may engage a CPA to review its internal controls, evaluate accounting software, obtain tax advice, and so on.
Some businesses utilize accountants who are not certified, but are very experienced and effective. You could ask your banker to recommend an accountant who works well with clients such as yourself.