BOARD MEETING DATE: September 21, 2001 AGENDA NO. 16

Health & Safety Code Section 40453 requires the AQMD to contract with a qualified firm once every three years to conduct a performance audit of the AQMD’s activities. The first audit was awarded in July 1991; the second audit was awarded in July 1994 and the third audit was awarded in July 1997.

An evaluation panel selected Thompson, Cobb, Bazilio & Associates based on an RFP issued in April 2000. The evaluation panel consisted of an attorney representing the California Manufacturing and Technology Association Southern California Air Quality Alliance; Chief of the Administrative Services Division, California Air Resources Board; and three AQMD staff made up of one Assistant Deputy Executive Officer, the Chief Financial Officer, and a Financial Analyst. At the June 2000 meeting, the Board approved a contract with Thompson, Cobb, Bazilio & Associates to conduct the performance audit at a cost not to exceed $139,860.

The objective of the performance audit was to evaluate AQMD policies, procedures, and productivity. The evaluation took into consideration the findings of the performance audit done in 1997 and recommendations adopted thereafter. The activities within the AQMD Work Program for FY ending June 30, 2000 were the foundation for the performance audit.

The performance audit was conducted in accordance with Generally Accepted Governmental Auditing Standards (GAGAS), the guidelines set forth in a statement issued by the Comptroller General of the United States entitled, Government Auditing Standards, Standards for Audit of Governmental Organizations, Programs, Activities and Functions. In the context of GAGAS, the performance audit focussed on reviewing effectiveness in achieving program results, economy and efficiency of operations, and compliance with laws and regulations.

Attached is Thompson, Cobb, Bazilio & Associates’ report on the Triennial Performance Audit of the AQMD. The Executive Summary includes a summary of the report’s recommendations. A total of 16 new recommendations were made in the areas of permit processing, information management, pollution control achievements and compliance with laws and regulations.

Achieving Program Results: The report highlighted the outstanding job the AQMD is doing in controlling the primary stationary sources of air pollution throughout the South Coast Air Basin. It notes that when combined with the mobile-source pollution control efforts of the CARB, the AQMD’s pollution control measures have continued to yield substantial improvements in air quality. The report recommends that the AQMD should take a lead role in developing a credit program to fund and eliminate super mobile polluters (cars and trucks) in the basin. It also recommends a rebalancing of the AQMD’s Scientific, Technical, and Modeling Advisory Group to include more representatives from the academic community.

Economy and Efficiency of Operations: The report reviewed the actions taken to address the recommendations from the prior audit and noted that while significant progress has been made, continued efforts in the areas of organization structure, use of facilities, performance measures, check handling and refunds should be pursued to achieve the anticipated benefits. In the area of Information Management, the auditor’s recommended the implementation of "Service Level Agreements" to document user expectations; an increase in the upgrading cycle of user workstations from 25% to 33% per year; specific hardware and software upgrades to improve the performance of the systems which handle permit processing, inspection and billing operations; the analysis and documentation of the business process workflow for permit processing; and the development of a formal disaster recovery/contingency plan for information systems.

Due to the high priority placed on permit streamlining and the reduction of the permit application backlog, the auditors spent a substantial amount of time testing several of the new permit streamlining measures and systems to identify processes that can be further enhanced. The report made several recommendations to help improve the processing of permit applications such as the development of more definitive guidelines on when to reject applications; the need to expand our efforts in marketing the benefits of our Streamlined Standard Permits; a re-evaluation of our Certified Permit Processor program to determine if the program still provides a benefit; further refinements to our Permit Administration and Application Tracking System (PAATS) to further increase its effectiveness; continued aggressive efforts to prioritize applications based on severity of emissions, location, and potential health risk; and improved tracking of applications to ensure that prescreening of applications is performed on a more routine basis.

Compliance with Laws and Regulations: The report found, based on a review and evaluation of AQMD programs, that the AQMD was in compliance with the exception of the Air Toxics "Hot Spots" program. The report noted that the AQMD has made significant progress towards completing program requirements, however, some program elements were not completed in accordance with deadlines contained in state law. The auditor’s recommended that the AQMD should continue to streamline the process for review and approval of the remaining Phase II HRAs.

Staff has prepared the attached responses to the audit recommendations and in many instances has begun to implement the recommendations. As required, the report, along with the AQMD’s action plan to implement the recommendations, will be submitted to CARB and the state legislature.

Of the 13 prior audit recommendations, the AQMD has implemented all, with some additional action needed in the following areas:
Organizational Structure – AQMD has made significant progress in this area. Management should continue to keep a close vigil to be sure that the balance of line to staff is maintained into the future.	*AQMD Response:* Significant "flattening" of the organization has occurred. However, the unique nature of AQMD and the type of technical oversight of line work that is required need to be taken into account when considering any further increases in supervisor/manager span of control.
Use of Facilities – The AQMD should relocate staff to the 3^rd floor and aggressively move forward to lease the ground floor space made available. AQMD should also reevaluate existing office space with the help of a space planner to possibly gain additional efficiencies and free more space for leasing.	*AQMD Response:* AQMD has aggressively pursued facility leasing and currently has no space available to lease. The cost-effectiveness of creating additional space for leasing by completing the 3^rd floor for use by staff will be pursued in the near term and will include consideration of costs for additional storage space and of meeting parking space requirements.
Performance Measures - AQMD should continue the first steps taken in the Permit Processing unit to develop meaningful performance measures. All staff should be made aware of the process and the active participation of people on the line that deliver the services should be solicited so that fair, yet meaningful results- based measures are developed and used to help guide the agency in a program of continuous improvement. While our focus was in the Permit Processing unit, similar steps should be taken throughout the organization. As an alternative, a pilot program can be conducted with the permitting unit, continuing on with the initial steps taken. Later, as the agency learns from this experience the process can be expanded to other units.	*AQMD Response:* The AQMD implemented a program tracking system (work program) in 1989 that accounts for expenditures and personnel at the program activity level. Staff will initiate a pilot program, building upon measures already implemented in permit processing as part of permit streamlining, to develop a meaningful results-based performance measurement system.
Cash Handling – The District should make the necessary formal written revisions to the Check Handling Procedures to reflect process changes that have taken place with the implementation of PAATS.	*AQMD Response:* AQMD’s written procedures regarding Cash (Check) Handling Procedures have been revised to incorporate the PAATS process changes.
Customer Service – Ensure that any open systems programming and processing issues be corrected as expeditiously as possible, concurrent with the change in Refund personnel.	*AQMD Response*: Information Management and Finance will continue to meet to resolve all open systems programming and processing issues. Funding for outside programming support is included in the FY 2001-2002 budget. An analysis of the refund process was completed in February 2001 resulting in the streamlining of the process.

In an effort to help expedite the processing of permit applications and provide a follow-up mechanism to obtain information from applicants, the use of student interns should be considered an option.	*AQMD Response*: In the past, AQMD has used student interns under direct supervision to help obtain additional information to assist in processing permits. Staff continues to support the use of student interns to help process permits.
AQMD management should also provide more definitive guidelines and implement a policy on rejecting incomplete permit applications.	The AQMD has a certain set of definitive policies for rejecting permit applications and the more common ones are hard-coded in the Permit Application Administration Tracking System (PAATS) program. However, in an effort to be more customer- friendly, AQMD has attempted to work with the businesses to obtain the necessary information, rather than rejecting all incomplete applications. To improve the quality of information received for a permit application, AQMD is selecting a contractor to redesign the permit application forms and supplemental information forms. Nonetheless, Staff will further define criteria for appropriate rejection of applications.
In recognition of the Task Force expectation that Streamlined Standard Permits offer an exceptional benefit for reducing the permit backlog, AQMD should refocus and step-up its efforts, especially in marketing and making interactive streamlined standard permits available on the Web. The establishment of performance measures would be a useful tool to help focus efforts upon the desired outcomes and for determining customer satisfaction.	*AQMD Response:* The existing Streamlined Standard Permits were developed in consultation with and outreached to representatives from trade associations. In addition, a brochure on the program was printed and information on the Streamlined Standard Permit was published in District newsletters. Staff from the Small Business Office were trained and educated on the program and their outreach efforts resulted in several applications being filed. To create an incentive to increase participation in this program, a new lower fee for Streamlined Standard Permits was incorporated into the recent amendments to Rule 301. Staff is working to expand the type of equipment that can qualify for a Streamlined Standard Permit and agrees that a more aggressive marketing campaign and use of methods to assess customer satisfaction will be beneficial once the additional packages are developed.
AQMD management needs to perform a thorough re-evaluation of the CPP Program’s mission, purpose and future and decide whether there is a need to continue the existence of this program.	Even though the CPP option only sees limited use, AQMD Staff continues to support the existence of the program and administers the CPP examination once a year to qualify additional CPPs. Staff provides on-going support to CPPs requesting access to AQMD's New Source Review data upon furnishing proper authorization from their respective clients. AQMD continues to flag CPP applications. AQMD also holds informational meetings for CPPs to update them on new rules, regulations, and programs. AQMD has provided additional assistance to CPPs and other applicants by posting guidance and samples of our engineering evaluations for permits and our policies and procedures on AQMD's website. Periodic updates to the AQMD's website to enhance the CPP program are done on an ad-hoc basis as well as holding informational sessions to the CPPs on new rules and regulations and programs. AQMD will continue to monitor and reevaluate this program.
AQMD management should aggressively continue to pursue further system improvements and refinements of PAATS to increase its effectiveness.	*AQMD Response:* AQMD agrees and has identified and planned for major enhancements to the Permit Application Administration and Tracking System (PAATS) which are scheduled for FY 2001-2002. AQMD will continue to make improvements and enhancements to increase its functionality and to make it more user-friendly.
AQMD management should ensure that an active Total Quality Management System is established as expeditiously as possible.	*AQMD Response:* Several programs have been implemented that integrate TQM principles to continuously measure and improve efficiencies and performance of the permitting systems. These include setting goals and expectations for each permitting Team, monitoring performance measures of each permitting Team, and providing management with tools to generate reports to track progress and achievements of performance measures. AQMD will continue to work on further development and enhancements of a Total Quality Management System.
The AQMD’s continued aggressive efforts to prioritize and process applications based on the severity of emissions, location, and potential health risks should be encouraged.	*AQMD Response:* AQMD has prioritized processing of applications for Permits to Construct new or modified equipment, as well as applications for Permit to Operate existing equipment which does not have a prior Permit to Construct. In most cases, when the application is submitted due to an enforcement action, the inspector would have either issued a Notice to Comply or a Notice of Violation if it was determined that the equipment was operating in violation of a rule. In these situations, the applications are assigned a higher priority, especially if there is Variance or an Order for Abatement hearing pending. However, with the step up in compliance activities, the requirements for emission inventory updates for AB 2588 (Toxic Hot Spots Program) and for Emission Fee Billing, the high risk facilities or operations are inspected and identified for priority processing. AQMD will continue to explore means to identify and expeditiously process applications for sources which may have higher toxic emissions or potential health risks. AQMD also expedites processing of applications to install control equipment for any source which may pose an environmental risk. Staff will also propose greater consolidation of the current two-step permitting process into a one-step combined PC/PO.
Section managers should be reminded to follow up with permitting staff and ensure that pre-screening of applications is performed on a more current and/or routine basis.	*AQMD Response:* During the weekly manager's meeting, production goals and achievements are discussed and new goals and expectations are set. The PAATS has been modified to enable the managers and/or supervisors to obtain current information on applications that are pending pre-screening that are assigned to their respective teams. AQMD has prioritized pre-screening and with the further development of PAATS will continue to improve pre-screening and processing of permits.
Also, in concert with IM, efforts should be pursued to ensure that tracking data between PAATS and website data is updated, both accurately and timely.	The data generated in CLASS from PAATS is the same data that is updated in the web application tracking system, as both programs access the same data tables in the database system. AQMD will continue to monitor this to ensure timely updates of this.
AQMD management should encourage that the PSTF diligently convene on a quarterly basis and issue updates on the status of progress until such time as the implementation process is complete. Similarly, management should also encourage compliance with the annual reporting requirements to the AQMD Governing Board.	*AQMD Response:* Although the initial recommendation was to provide quarterly PSTF meetings, these meetings have been scheduled on an as-needed basis, based on the recommendation of the PSTF. As noted in September 1999 and June 2000, such meetings were scheduled as per PSTF Chairwoman’s and Task Force Members’ request. The next meeting is scheduled for August 24, 2001. Although a separate annual report was not prepared, an annual report on permit processing is prepared by AQMD and presented to the Governing Board pursuant to the provisions of SB 1928. This report summarizes the permit activities for the calendar year and is categorized by different industry SIC codes. The last annual report was prepared and submitted to the Governing Board on May 11, 2001. AQMD will prepare a separate annual report on permit streamlining to provide progress status to the Board.

IM should take further steps and implement a "Service Level Agreement" to document expectations and levels of service. In addition, IM should utilize the Intranet to disseminate pertinent information to the Users.	*AQMD Response*: AQMD Administrative Policy and Procedures 41-504 for Information Management contains guidelines that establish the use of a scoping statement and impact report to initiate software development. IM will evaluate this policy to determine if it needs to be expanded.
The current Users’ workstation upgrade should be continued and, if possible, increase the upgrade rate to 33%. It should be noted that at the current rate of 25% per year, it will take 4 years to completely upgrade all users. According to the experts, the approximately useful life of a PC/Workstation is 3 years because of the constantly improving hardware and software. As a result, at the current rate of 25%, AQMD would be continuously upgrading their workstations.	*AQMD Response:* Users’ workstations will continue to be upgraded at a minimum of 25 percent a year. The rate of future upgrades will be addressed in the next revision to the Information Management Strategic Plan and the FY 2002-2003 Budget.
To improve CLASS system performance IM should:
Implement the planned cabling upgrade from the Users workstation to the switches closet as soon as possible. The current 10 MB copper cable should be upgraded to 100 MB fiber optic cable.	*AQMD Response:* Network cabling will be upgraded from 10 MB to 100 MB during Fiscal Year 2001-2002.
Implement the planned Server CPU upgrade. The current 300 MHz, 1.5 GB RAM CPUs should be replaced with 700 MHz, 11 GB RAM CPUs.	*AQMD Response:* The Server CPU and Memory upgrade is underway and should be completed by August 2001.
Upgrade the database software by implementing Ingress II database software.	*AQMD Response*: The database software version upgrade to Ingress II was completed in June 2001 and is undergoing fine-tuning.
The AQMD IM Office, working jointly with the individual AQMD user organizational units, should analyze and document the business practice workflow across functional lines for the permit granting processes. A permit processing business workflow would identify key processes and assist AQMD in determining who should perform the process. Each functional area would be aware of how their input affects the entire process.	*AQMD Response*: IM continues to develop Business Process Models for all new and enhanced system development projects. Both the contractor hired to conduct the Permit Streamlining Study and the contractor hired subsequently to analyze AQMD's permitting systems documented the business process and practices. However, these need to be updated to reflect the permit streamlining changes that have been made subsequent to these studies.
The AQMD develop, test, and implement a disaster recovery and contingency plan for its operations. At a minimum, the plan should be updated yearly or when any major component of the information system enterprise changes.	*AQMD Response*: A disaster recovery/contingency plan will be developed by June 2002.

Rebalance the composition of the Scientific, Technical, and Modeling Advisory Group to include more representatives from the academic community. It would be helpful if the science-based advisory groups had qualified scientists and engineers from recognized educational institutions such as Caltech, UCLA, UCR, USC, and UCI that have expertise in this area.

AQMD Response: When the AQMD expanded its Advisory Council, the AQMD Governing Board formed two advisory groups to follow the development of future air quality management plans (AQMPs), the "AQMP Advisory Group" and the "Scientific, Technical and Modeling Peer Review Advisory Group" (STMPRAG). The AQMP Advisory Group is a policy-oriented advisory group with the primary charter of providing comments on the development of future control strategies. However, in order to provide meaningful comments on control strategies, the group needed to have members who have technical expertise in emissions inventory and attainment demonstration development. To provide further technical support, the STMPRAG was formed with experts from the academic community, environmental community and private sectors. The technical expertise of the STMPRAG range from the fields of atmospheric chemistry, photochemical model development, air monitoring, meteorology, economic analysis, and medicine.

The STMPRAG was formed as an outgrowth of comments received from the "Modeling Working Group" and the "Socioeconomic Assessment Working Group." The two groups are linked in their efforts under the AQMP development process. Because of this link, each group independently expressed their frustrations in not having the ability to fully interact with each other on an ongoing basis. The STMPRAG provides a forum for the air quality technical experts to communicate with the economic experts in understanding the full scope of analysis conducted in preparing the AQMP. In addition, a medical expert was appointed to the Advisory Group to broaden the Group’s breadth.

As indicated in the report, there is an overall balance between the air quality and socioeconomic experts. However, the AQMD staff believes that all of the air quality experts have experience in air quality modeling. While there are only a limited number of members in the academic field at this time, the AQMD staff will strive to seek input and participation from academic experts as future plans are developed.

As part of the 2001 AQMP revision, CARB has retained two professors (Dr. Rob Harley, UC Berkeley and Dr. William Carter, UC Riverside) to make initial review of the computer modeling efforts to date. In addition, the AQMD will retain technical experts to conduct independent peer review at the time the AQMP is released for public review.

EPA and CARB should partner with the AQMD to grant the District a lead pilot role in developing a credit program to fund and eliminate super mobile polluters in the basin. Funds should be advanced by AQMD to support an economic analysis performed by a research organization or educational institution to develop the mobile credit program.

AQMD Response: The AQMD has Rule 1605 – Credits for the Voluntary Repair of On-Road Motor Vehicles Identified Through Remote Sensing Devices, provides a mechanism for the generation of credits through remote sensing. Any type of remote sensing activity where credits are generated must be planned in such a manner that credits generated are not considered double-counted such as with the State’s Smog Check II Program.

The Auditor’s recommendation will be forwarded to the California Air Resources Board for consideration of inclusion in the State Motor Vehicle Pollution Control Plan currently under development.

South Coast Air Quality Management District