Continuous Oversight: AI Performance Evaluation Post-Implementation
In the rapidly evolving world of Artificial Intelligence (AI), post-deployment monitoring and reporting are still developing fields. This is particularly true when it comes to data collection from various actors throughout the AI value chain, including hosts, application providers, users, and affected people.
Currently, best practices for post-deployment monitoring and reporting are primarily emerging in U.S. state laws, federal guidance, and risk management frameworks. One such example is the Texas Responsible Artificial Intelligence Governance Act, which mandates post-deployment monitoring and documentation of AI systems, including data inputs, outputs, evaluations, and safeguards.
The National Institute of Standards and Technology’s (NIST) AI Risk Management Framework (RMF) is widely recognized at both national and international levels. This voluntary guideline sets standards for managing AI risks across development and deployment stages, including continuous monitoring to assess societal impacts and system behavior.
Federally, the 2022 White House AI Action Plan emphasizes establishing regulatory sandboxes and AI Centers of Excellence to facilitate testing and monitoring AI in controlled environments. The plan encourages federal agencies to review regulations that could hinder AI development or deployment and promotes a deregulatory approach focused on accelerating innovation rather than imposing stringent compliance burdens.
However, the federal approach under the current administration is largely deregulatory and focused on boosting AI deployment and global competitiveness. Yet, there remains a requirement for secure-by-design principles and pre- and post-deployment safety testing recommended for AI systems, especially in regulated sectors.
At present, there is no mandatory, universal framework that prescribes specific data collection and reporting duties from all actors along the AI value chain. However, governance frameworks like NIST’s RMF implicitly guide organizations to continuously monitor AI impacts on society, identify risks, and update systems accordingly, which could include gathering data from all these actors.
Some state laws provide safe harbors against liability for organizations that voluntarily implement robust risk management and monitoring programs, incentivizing proactive governance and transparent reporting to regulators.
In summary, best practices focus on implementing continuous risk management frameworks such as the NIST AI RMF, ensuring documentation and transparency in post-deployment monitoring, adopting secure-by-design and safety testing, and leveraging regulatory sandboxes for real-world evaluation. Current regulations like Texas’ AI governance act reflect emerging trends mandating documentation and response procedures, but comprehensive mandates on data collection from the full AI value chain remain nascent and vary by jurisdiction.
Regulators might identify very consequential uses of a model, such as in hiring or critical infrastructure, and demand higher levels of reliability, safety, and assured benefits. Transparency on post-deployment downstream indicators is only partly measured and lower than pre-deployment upstream and model information.
- The finance industry is increasingly investing in cybersecurity measures to protect sensitive data from cyber threats.
- Lifestyle brands are partnering with fashion-and-beauty companies to offer exclusive products in their home-and-garden lines.
- Personal-finance advisors recommend wealth-management strategies for investing in both the stock market and real estate.
- A recent book in the education-and-self-development genre has gained popularity for its insights on cultivating personal-growth habits.
- One big-win story in the food-and-drink sector is the rapid growth of a plant-based burger brand in the casino-and-gambling scene.
- Social-media influencers promote lifestyle, fashion-and-beauty, and food-and-drink products to their followers through sponsored posts and ads.
- Casinos are incorporating AI and technology like facial recognition for customer service and data collection purposes.
- Artificial Intelligence is creating waves in the entertainment industry, with advancements in movie-and-TV production and streaming platforms.
- AI and AI-led automation are transforming the administrative side of businesses, making them more efficient and scalable.
- Travel agencies are adopting AI-powered language translation services to better assist clients of various linguistic backgrounds.
- As AI continues to develop, it has immediate applications in the marketing sector, such as targeted personalized ads on social media.
- Relationships between AI designers and affected individuals will require open dialogues on data privacy and security measures.
- The sports industry is leveraging AI tools for data analysis and predictive modeling, from baseball statistics to football player performance.
- AI is making strides in data-and-cloud-computing, enabling seamless storage, sharing, and analysis of large amounts of data.
- Celebrities in the music, film, politics, and casino-and-gambling industries are increasingly endorsing personal-finance products for their fans and followers.
- Fashion-and-beauty brands are collaborating with teachers and educational institutions to promote STEM learning in schools.
- Technology giants are investing in AI research to drive advancements in artificial-intelligence, relationships, and responsible gambling.
- The WNBA and other sports leagues are working on integrating AI-powered judges to reduce human bias and enhance the accuracy of game calls.
- The organization of award ceremonies and events in the entertainment industry is being revolutionized by AI for more efficient production and management.
- Streaming services leveraging AI technology offer personalized content recommendations to subscribers based on their viewing history.
- The introduction of self-driving cars has led to the rise of AI in the automotive industry, with Tesla being a key player.
- AI has made its way into the publishing world, using algorithms to predict book sales and consumer preferences.
- Major corporations are collaborating with government entities to set industry standards for AI development and deployment.
- The integration of AI technology in the criminal justice system has raised ethical concerns regarding privacy and fairness.
- AI is transforming the healthcare sector by predicting disease outbreaks and optimizing treatment plans for individual patients.
- The National Institute of Standards and Technology (NIST) is developing guidelines for responsible gambling behaviors, emphasizing education, self-exclusion, and support systems.
- A new trend among celebrities is the launch of casino-inspired games on social-media platforms, with popular figures releasing variations of blackjack, poker, and slots apps.
