> ## Documentation Index
> Fetch the complete documentation index at: https://docs.atomscale.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Set Up Tool Logs

> Configure the Tool Log Streamer to stream CSV or EpiTrend logs from a watched folder

This page sets up the **Tool Log Streamer** integration. Before you start, check that your logs meet the [requirements](https://docs.atomscale.ai/gateway/tool-state.md#requirements-for-csv-logs).

## Before You Start

* A gateway with read access to the folder where the tool's control software writes its logs.
* For CSV logs, a sample log file from your system to draft and check the parsing schema.

## Set up with the assistant

Open the [Gateway assistant](https://docs.atomscale.ai/gateway/assistant.md#open-the-assistant) and choose **Stream tool logs** in a new chat, or write your own request. Give it the log folder and format. For CSV logs, use a sample to draft and check the schema before applying it:

1. Give the assistant the full path to a representative CSV log on the gateway. For example:

   > Set up Tool Log Streamer for CSV files in `D:\MBE\Logs`. Use `D:\MBE\Logs\sample.csv` to draft a parsing schema. Keep substrate temperature in °C and chamber pressure in Torr. Show me the preview before applying the schema.

   If the assistant can't read the file, paste the start of a sample you can share. Include any preamble, the header, and several data rows.
2. Check the parsed timestamps, channel names, units, and any rejected rows in the preview. Ask the assistant to correct any wrong assumptions and run the preview again. Previewing doesn't change the integration.
3. Ask the assistant to apply the schema, then approve the change. Once the configuration is saved, start the integration when you're ready. When the tool writes a new log, [plot a channel to check that data arrives](#csv).

EpiTrend logs don't need a parsing schema. Give the assistant the `EpiTrendData` folder instead.

## Set Up the Integration

To configure Tool Log Streamer yourself, follow the steps for your log format.

> [tab] CSV
>
> **1. Add the integration**
>
> Open Integrations, click Add Integration, and click
> Add on **Tool Log Streamer**. Add one copy per log folder.
>
> **2. Choose the format and folder**
>
> The configuration page opens after you add the integration. If you added it during
> [First-Run Setup](https://docs.atomscale.ai/gateway/first-run.md), click it on **Integrations** and click
> Configure. Leave **Format** on **CSV**, and
> set **Watch Path** to the folder where logs are written, with
> Browse or an absolute path.
>
> **3. Adjust the folder scan if needed**
>
> The defaults watch `.csv` files up to two subfolder levels deep, and end a file's capture
> after it hasn't grown for 600 seconds. If your logs differ, turn on
> Show advanced settings and adjust **Traversal Depth**, **Filename Regex** (for files that don't end in
> `.csv`), or **Inactivity Timeout (Seconds)**. The timeout must be longer than the longest
> normal pause between writes during a run.
>
> **4. Add a parsing schema**
>
> [Ask the Gateway assistant to draft and check a schema](#set-up-with-the-assistant),
> or add one yourself. Under **Parsing schema**, click Browse to load a schema file,
> or Create new parsing schema to start from the built-in
> example. Edit it so the column names match your log's header exactly. See the
> [Parsing Schema Reference](https://docs.atomscale.ai/gateway/tool-state/parsing-schema.md).
>
> **5. Apply and start**
>
> Click Apply Configuration. Atomscale checks the schema and
> saves a copy with the integration. Then click
> Start Integration.
>
> **6. Verify with a run**
>
> When the tool software starts a new log, a **Tool Logs - …** data item starts uploading as
> Stream Active. Its chart starts empty: right-click the chart, or click its gear, and click
> \+ next to a channel to plot it. The chart then fills in as rows are written.

> [tab] EpiTrend (Veeco MOLLY)
>
> The gateway must run on the MOLLY computer, because the integration ends each hour by the
> computer's own clock.
>
> **1. Add the integration**
>
> Open Integrations, click Add Integration, and click
> Add on **Tool Log Streamer**.
>
> **2. Choose the format**
>
> Set **Format** to **EpiTrend**. This sets **Filename Regex** to
> EpiTrend's hourly files and **Inactivity Timeout (Seconds)** to 120. No parsing schema is
> needed.
>
> **3. Point it at EpiTrendData**
>
> Set **Watch Path** to MOLLY's `EpiTrendData` folder. Its year and month subfolders are found
> automatically.
>
> **4. Apply and start**
>
> Click Apply Configuration, then
> Start Integration.
>
> EpiTrend writes one pair of files per hour, such as `23day-10hr.txt` and
> `23day-10hr-binary.txt`, and rewrites both as the hour goes on. The integration takes a snapshot
> at most every **Minimum Snapshot Interval (Seconds)** (10 seconds, advanced) for the live
> chart. Each hour is its own data item; it
> completes once the next hour starts and the old files have been unchanged for the inactivity
> timeout, and the hour's two files are then attached as its raw files.
>
> Only the current hour and later are captured. Earlier hours are not uploaded, and **Upload
> Existing Files** has no effect.

> [!NOTE]
>
> By default, CSV files already in the folder when the integration first starts are not uploaded.
> To backfill them, turn on Show advanced settings and Upload Existing Files before the
> first start. Existing files that
> grow later are picked up either way.

## Working with the Parsing Schema

The schema control shows the active schema with Preview Schema and Edit Schema views. In the editor, Format JSON tidies the text, and **How to write a parsing schema** summarizes the rules.

* Loading a file takes a snapshot. Apply Configuration validates and saves that snapshot. Editing the source file afterward doesn't change the saved schema; load it again and apply.
* Each data item keeps the schema that was active when its file started. A schema change applies to new files only.
* The schema is saved with this integration on this gateway. Import configuration doesn't include it, so keep your schema file somewhere safe.

> [!TIP]
>
> To test a CSV schema without the tool, turn on Show advanced settings, set **Source** to
> **Synthetic (Testing)**, apply your schema,
> and start the integration. It writes a test CSV with one row per second from the schema's columns.
> Synthetic data needs date-time timestamps (not Unix seconds), and doesn't write recipe columns.
> Switch back to **Files** when you're done.

## Settings Reference

Only **Format**, **Watch Path**, and the parsing schema show by default. Fields marked advanced appear with **Show advanced settings** turned on.

| Field | Default | Description |
| - | - | - |
| **Source** (advanced) | Files | **Files** watches a folder. **Synthetic (Testing)** generates CSV test data from the parsing schema. |
| **Format** | CSV | **CSV** or **EpiTrend**. Changing it resets **Filename Regex** to that format's default. |
| **Watch Path** | None | The folder containing the logs. Must be an absolute path to an existing folder. |
| **Traversal Depth** (advanced) | 2 | Subfolder levels to watch, from 0 to 64. 0 watches only the folder itself. |
| **Filename Regex** (advanced) | `(?i)\.csv$` | A regular expression matched against each file name, including its extension. |
| **Upload Existing Files** (advanced) | Off | CSV only. Uploads files already in the folder on first start. |
| **Inactivity Timeout (Seconds)** (advanced) | 600 (120 for EpiTrend) | Seconds without growth before a file's capture ends, from 1 to 86,400. Data written after that isn't uploaded. |
| **Polling Interval (Seconds)** (advanced) | 1 | How often to check the logs for changes, from 1 to 60. |
| **Minimum Snapshot Interval (Seconds)** (advanced) | 10 | EpiTrend only. Minimum time between snapshots of the current hour's changed files, from 1 to 3,600. |
| **Parsing schema** | None | CSV only, and required. Tells Atomscale how to read the log. |
| **EpiTrend Diagnostics** (advanced) | On | EpiTrend only. Records extra detail about file changes for support. |
| **Activity Summary Interval** (advanced) | 10 minutes | How often a summary is written to the activity log. |
| **Upload Backlog Size Limit**, **Free Disk Space To Keep** (advanced) | Auto, 1 GB | See [Disk Buffering](https://docs.atomscale.ai/gateway/uploads.md#disk-buffering). |

### Example Filename Patterns

| Goal | Filename Regex |
| - | - |
| Any `.csv` file (default) | `(?i)\.csv$` |
| Only files starting with `GrowthLog` | `^GrowthLog.*\.csv$` |
| Tab-separated `.txt` logs | `(?i)\.txt$` |

## Fixing a File That Failed

Parsing problems don't stop the raw upload, but a file that failed parsing can't be fixed in place:

1. Stop writing to the problem file.
2. Stop the integration, correct the parsing schema, and apply it.
3. Start the integration again and start a new log file.

Only new files use the corrected schema. The original file's raw bytes stay stored in its data item.

## Troubleshooting

Check the integration's **Activity Logs** on the gateway, and the data item's status in the web app.

| Message or symptom | Cause and fix |
| - | - |
| Provide an explicit parsing schema in configuration before starting this integration. | No schema has been applied. Load or create one and apply the configuration. |
| The saved parsing configuration is for a different Format. | **Format** changed after the schema was applied. Apply a schema for the selected format. |
| Apply Configuration rejects the parsing schema | The schema is invalid. The message names the problem; see [Validation Rules](https://docs.atomscale.ai/gateway/tool-state/parsing-schema.md#validation-rules). |
| Watched folder is missing or inaccessible: … | **Watch Path** doesn't exist or the signed-in Windows user can't read it. For network shares, check the share is connected for that user. |
| Source access delayed for … retrying automatically | The tool software holds the file locked. The integration retries on its own. |
| Schema columns absent from header | The schema's column names don't match the CSV header exactly, including recipe columns. Correct the schema and start a new file. |
| Timestamp has no UTC offset and the lab computer's time zone is unknown | The log has no UTC offset and the gateway didn't report its time zone. Update the desktop app, or write timestamps with an offset. |
| Timestamp decreases relative to the previous record | Rows went backward in time, for example after the tool computer's clock was set back. |
| CSV changed after capture ended | Data was written after the inactivity timeout. Increase **Inactivity Timeout (Seconds)**, and write each run to a new file. |
| Session has a data gap | The file was truncated or replaced during capture. A new data item starts for the new content. |
| A deleted and recreated file uploads again | A file recreated at the same path, or old logs copied into the folder, count as new files. |
| Many values are blank | Cells that aren't numbers are stored as missing. Check the schema's `decimal` separator, and list text columns in `drop_columns`. After 50 consecutive unreadable rows, parsing stops with an error. |
| A tool log doesn't appear next to its growth | Link the Tool Log Streamer and the RHEED camera to the same **Growth Instrument**. See [Finding Your Data](https://docs.atomscale.ai/gateway/data.md). |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.