How to use open-source AI safely in your organisation

The advantages of open source apply to AI as well. An open model can run in your own environment. The code can be inspected and adapted. The work a supplier would otherwise do falls to you: choosing the version, checking where it comes from and following up when something changes.

Agents and add-ons from unknown developers

Personal AI agents running on an employee’s computer can read and write files, drive the browser, send email and run commands. Installed on an employee’s own initiative, such an agent can reach everything they can. In April 2026 France’s CERT-FR stated that such agents should not be used on work computers and mobile devices until the products are stable and proven for security. Until then it recommends isolated test environments without sensitive data, prior approval by IT and security, and mandatory human approval for system commands.

The agents become more useful with add-ons, such as connections through MCP or ready-made skills from a marketplace. Depending on how the agent is built, an add-on can run as trusted code with the same permissions as the agent itself. In early 2026 security researchers went through the marketplace for a popular open-source agent and found hundreds of malicious skills. Several had names only a letter or two away from the marketplace’s official tool.

Misspelt names are an old trick in open package registries, and AI has added a new twist. When researchers had coding models write software, the models sometimes suggested packages that did not exist, and the same invented names often came back. Whoever registers such a name can then wait for someone to install it. An agent that installs whatever the model suggests can make that risk greater. How an agent can also be misled by hidden instructions is covered in the risks of AI agents.

Updates nobody has reviewed

A component that was safe yesterday can be harmful today. In August 2025 malicious versions of a well-known build tool were published to an open package registry. Available for a few hours, they searched developers’ computers for sensitive data, tried to enlist AI tools already installed and uploaded what they found to public code repositories.

That same summer, malicious code found its way into an official release of an add-on for coding assistance, after an access token in the build pipeline had been scoped too broadly. The code contained an error and could not run, and the release was withdrawn. Both cases argue for reviewing new versions before they are let in, pinning versions and keeping automatic updates away from computers that hold sensitive data.

Model files and where they come from

Pickle is the default format for model weights in one of the most widely used machine learning frameworks, and a pickle file can run program code the moment it is loaded. In 2024 researchers found around a hundred models containing malicious code on a large model platform. In early 2025 models with malicious code were found that the platform’s scanning had not flagged. They were removed within a day, and the scanning was improved. Formats that hold only the model’s weights, such as safetensors, reduce the risk.

OWASP notes that published models carry no strong assurance of origin, and that a model card offers information but no proof. An account can be taken over, or a look-alike account created. A model may also have been altered to behave differently in certain situations, unnoticed by ordinary tests. The same applies to the smaller adapter files that tailor a model to a particular task.

Rules for what gets downloaded

Guidelines from 2023, developed by the NCSC with CISA, expect suppliers of AI components to meet the same security standards the organisation applies to other software. In practice that can mean an allow-list of approved open models, packages and add-ons, review of new versions before they are used, and running anything not yet reviewed separately in a sandbox.

An inventory of every component, an SBOM, shows quickly where a vulnerable version is in use. For AI it also covers models, datasets and their origin. In May 2026 the G7 Cybersecurity Working Group published guidance on the minimum such an inventory for AI should contain. Checksums and signatures show that a file is unchanged and who published it. Whether its contents are safe is a matter for the review. Each component also needs an owner, so that someone notices when a package changes maintainer or a supply chain attack becomes known.

How it works in iSearch

iSearch ships with locked dependencies and an SBOM, a list of all components. Updates are signed, including for fully isolated environments. The language model can run in your own environment, along with the rest of the chain. Every customer gets a separate installation of its own, and your questions and documents are never used to train models.

Read more about security in iSearch and about on-premises or cloud, or book a walkthrough.