Langium AI

Appearance

Quickstart ​

This guide will walk you through getting set up with langium-ai-tools and lai (the langium-ai CLI) in your existing Langium project. The end result will be a working integration with Langium AI, a language descriptor, generated system prompt, and initial stubbed out evaluations.

If you're just getting started with Langium AI, this is the right spot to be for getting set up.

Before you dive in!

Everything that's generated by the lai CLI can be refined & edited by hand. The predominant goal of the CLI is to make it easier to get started, but it's often the case that you'll want to iteratively improve on things from there.

Prerequisites ​

As a heads-up here are some base requirements you should meet before getting started.

  • Node.js versions 22 or greater
  • Langium 4.x or newer
  • ESM-importable services. Specifically your language module should export a createMyDSLServices function that can be pulled in via import.

Some preflight checks ​

The lai CLI is compatible with Langium projects that contain more than one project (such as the requirements example on the Langium project itself).

However, ensure there's just one langium-config.json reachable from where you run lai. We take this as a key file to help map out your project's structure.

Your services need to be ESM compatible. lai expects to be able to import your language's service functionality this way. This will cause issues during lai eval if your project isn't configured for ESM.

Ensure that you've built your project already, i.e. running langium:generate and build.

1. Install the CLI ​

bash
npm install -g langium-ai

The langium-ai package includes the lai CLI. Once that's installed, you can confirm it's available on your path:

bash
lai --version
lai --help

2. Initialize your project ​

From your project root, run init to set everything up. You'll be prompted along the way for generating files, as well as for installing langium-ai-tools for evaluations.

bash
lai init

In general, the initialization works through the following steps:

  1. Detects your project: It locates your langium-config.json, registered languages, core project module, and all custom services it can find (tokenizer, validation, LSP related services, etc.).
  2. Asks for a project name, defaulting to your package.json name. This name determines your descriptor and system prompt filenames.
  3. Offers to install langium-ai-tools into your project using the package manager it detects (npm or pnpm). This is required for evaluations to work.
  4. Scaffolds evals/. These are stubbed out evaluations that outline the basis for how evaluations are performed, along with helpful examples & utilities.

When it finishes you should have the following files:

  • lai.config.jsonc: the LAI configuration file
  • An evals directory with basic.eval.ts and utils.ts. These are stubbed evaluations & utility functions respectively.

Plus, the latest langium-ai-tools should be added to your project's dependencies.

For reference, here's what an example lai.config.jsonc config file looks like:

json
{
  "version": "0.4.0",
  "langium": {
    "configPath": "langium-config.json",
    "languages": [
      {
        "id": "minilogo",
        "grammarPath": "src/language/minilogo.langium",
        "caseInsensitive": false
      }
    ]
  },
  "descriptor": { "path": "minilogo.descriptor.yml" },
  "sysprompt": { "path": "minilogo.sysprompt.md" },
  "evaluations": { "directory": "evals" },
  "project": { "name": "minilogo" }
}

Non-interactive use

lai init -y (or --yes) skips every prompt and takes the defaults, such as the detected project name, and auto installs langium-ai-tools. Most lai commands have a -y option to allow scripting or CI usage.

Re-running just one part

If you find yourself wanting to regenerate just a part of your lai setup, lai init config re-detects your project and rewrites lai.config.jsonc without touching anything else. Similarly, lai init evals regenerates the evals/ templates only. Both accept the -y flag too. Both of these are destructive and won't preserve the existing file or directory they touch, so be sure this is what you want!

3. Generate a language descriptor ​

bash
lai gen descriptor

This writes up a descriptor for your DSL project, in the form <project-name>.descriptor.yml. This provides a structured YAML map of your DSL project for your grammar paths, services, validators, and up to three small example programs pulled from any examples/samples that are detected. It collects and organizes information about your project so later tasks, such as prompt generation, can be performed without having to hunt for everything again.

Although the generated version is helpful to getting started, it should be reviewed and edited by hand afterwards. No amount of automatic detection beats a manual refinement, but it certainly helps lessen the amount of time needed to do so.

At this point, no AI is involved. Descriptor generation is performed via static analysis of your project. With that being said, there's an lai-gen-descriptor skill for producing and/or further refining an existing descriptor. In practice, this can be quite helpful to account for services, examples, or details that weren't picked up automatically.

Regenerating over an existing descriptor

If the descriptor already exists, lai gen descriptor asks before overwriting — and that prompt is not suppressed by -y. If you do want it to proceed unblocked, use --fresh to overwrite unconditionally, such as in a script or CI.

4. Generate a system prompt ​

bash
lai gen sysprompt

This reads your descriptor, and writes <project-name>.sysprompt.md. This is a baseline system prompt which describes your language(s) to a model. It's also deterministic, and uses the precomputed mapping in your descriptor to lay itself out.

Currently, lai inlines validations by default up a certain size. This is crude but effective, and often benefits from hand-tweaking afterwards. However, if your validator is large, lai offers to emit a summarized map of its checks instead of inlining the source as is. This keeps implementation details slim, so it doesn't bloat the prompt.

Under -y, large validators are summarized automatically. As with the descriptor, --fresh overwrites an existing prompt without prompting.

5. Run evaluations ​

bash
lai evaluate

lai evaluate loads up the system prompt pointed to in your config, discovers .eval.ts files in the configured evals/ directory, and runs every case it finds.

For example:

ℹ Running 3 evaluation case(s) across 1 file(s)...

✔ basic.eval.ts: avg 100.0% (3 cases)
  ✓ Basic Code Generation Evaluation Examples > should generate a simple program (100.0%)
  ✓ Basic Code Generation Evaluation Examples > should match expected output similarity (100.0%)
  ✓ Code Explanation Examples > should explain code correctly (100.0%)

============================================================
Summary
============================================================
Total: 3
Ran: 3
Average score: 100.0%
Score range: 100.0% - 100.0%
✔ Results saved to: .langium-ai/... (Run #1)

The block above shows what a run might look like, but it's illustrative and does not show numbers reproduced from a test. Your suite names, case counts, and run path will be different.

Stubbed evaluations ​

If you open the default generated evals/basic.eval.ts, you'll find each case returns a constant before doing any work.

ts
const 
STUB
 = { 
score
: 1, 
stub
: true, 
note
: 'placeholder — not evaluating yet' } as 
const
;


evaluation
('should generate a simple program', async (
_ctx
: EvalContext) => {
    // STUB: returns a passing score without calling an LLM.
    return 
STUB
;

    // ── Real evaluation (uncomment and adapt) ──
    // ...
});

This is by design. Originally we wanted to fail early, but this turned out to be problematic when trying to verify whether everything was wired up correctly. It turned out that generating a passing default eval set as well as providing commented out instructions for how to do it for real were much more flexible, and less confusing when the default installation didn't pass on its first eval.

The real implementation sits below each return, commented out. It calls generateResponse() from evals/utils.ts which throws by default, because connecting a provider is something we leave to you. This aligns with one of the key goals of Langium AI, that we don't pick your model, but we do want to make it easy to support what you choose based on your needs.

The stub: true marker is carried into the saved run data, so a stubbed baseline is distinguishable from a real measurement in lai history, making it reasonable to sort it out later (assuming you don't clean your history first).

How scoring works ​

Every case returns a score between 0 and 1. Scores are roughly colored based on their value.

ScoreMarker
>= 0.8✓ green
>= 0.5~ yellow
< 0.5✗ red

Any case scoring below 0.5 makes lai evaluate exit non-zero, which is what makes it usable as a CI gate. Each run is persisted under .langium-ai/ with a sequential run ID, so it can be looked up (or tagged) later on. We recommend that you check in your run history so others can view & work based on your prior run results.

Useful flags while you're getting oriented ​

bash
lai eval --list        # list discovered files, suites, and cases without running them
lai evaluate --verbose # per-case detail as each one completes
lai status             # overall configuration status
lai history            # all prior runs, newest to oldest

You're all set ​

At this point you're ready to go. You've now created the config, descriptor, system prompt, evals, and a persisted run history. Everything from here expands further in each of these areas.

What's next ​

Troubleshooting tips ​

No langium-config.json files found Probably not inside a Langium project, or the file is missing. lai needs it to locate your grammar. Double check that you're running lai from the right spot.

Multiple Langium projects detected More than one Langium config file was picked up. Monorepos are supported, so long as there's only one config file. In the case where you have more than one separate Langium config file, run lai init from the specific language project you want to work on.

Detected Langium version 'x.y.z', which is not supported A heads-up warning. Langium AI expects Langium 4.x or newer. This will also set a non-zero exit code.

No registered Langium languages found in this project Your langium-config.json has no languages entries, or the grammar files it lists don't exist on disk. Most likely a corrupted config file or a mistake in auto-detection.

Descriptor not found at <path> Run lai gen descriptor before lai gen sysprompt.

System prompt not found. Run 'lai gen sysprompt' first.lai evaluate needs a system prompt to build the evaluation context. Generate it, or point to one you've provided with lai evaluate --sysprompt ./path/to/prompt.md. You can also update the lai.config.jsonc file's sysprompt entry to point to the default you choose.

An eval file fails to import Oftentimes this is a non-ESM issue (but not always). Double check that your project has been generated, built, and that the create<Name>Services import at the top of evals/basic.eval.ts is resolvable. The automatic generation may have picked up the wrong import path or function. lai init infers that path from your DI module and makes a best attempt to set this up correctly, but it can still get it wrong if your project layout is different in an unexpected way. You can also run lai eval --list to see import errors on their own, without running anything.