FCE Utilities and Diskover Task Management – FAQ

This FAQ provides guidance for managing and troubleshooting the File Classification Engine (FCE), Diskover Tasks, and related system components.

Unless otherwise noted, utilities are located in: /opt/recordpoint-connector-IndexBuilder/Deployment/

1. Utilities

1.1 configure-ova-rpms.sh

Q: When should I run this script?
Run this script after the initial FCE installation and immediately following any FCE upgrade using newer RPMs.

1.2 configure-static-ip.sh

Q: What does this script do?
If the virtual machine guest uses a static IP address, this utility configures FCE with the assigned static IP or hostname.

1.3 fcectl.sh

Q: What is the purpose of fcectl.sh?
This utility allows you to view and control FCE application services.

• Displays FCE runtime status
• Stops, starts, or restarts FCE services
• Run with -h or --help to display usage details

1.4 FCE.ConnectorApiTester/

Q: What is the Connector API Tester?
Inside this directory is a C# utility for testing connectivity to the RecordPoint Platform’s connector API. It can submit test records using the connector configuration defined in FCE.

2. Diskover Task Management and Cleanup

2.1 Naming Diskover Tasks

Q: Why does the task name matter?
Diskover task names become FCE Location Group names, prefixed with fce-.
Example: finance-smb-share01 → Location group fce-finance-smb-share01

Best Practices:

• Use unique, descriptive, and stable names
• Include source, scope, and environment (optional)
• Avoid generic names such as scan1 or sharedrive
• Suggested format: {department}-{share}-{pathOrScope}-{env}

Examples:

• finance-smb-share01-root-prod → fce-finance-smb-share01-root-prod
• eng-nfs-projA-designs-prod → fce-eng-nfs-projA-designs-prod

2.2 Stopping a Running Diskover Task

Q: How can I stop a running task?

1. In the Diskover UI, open the Tasks panel.
2. Use the dropdown for the running task and select Stop.
3. Refresh the page.
   If the task still appears active after 1–2 minutes, open the dropdown again and select Reset Status.

2.3 Disabling or Removing a Diskover Task

Q: What’s the difference between disabling and removing a task?

• Disable: Keeps the task configuration for reference or future reactivation.
• Remove: Deletes the task configuration and schedule but does not delete the indexed data in Elasticsearch.

To Disable a Task:

1. In the Diskover UI, open Tasks.
2. From the task dropdown, select Disable.
3. Disable the corresponding Location Group in FCE.

To Remove a Task:

1. In Diskover UI, open Tasks.
2. From the dropdown, select Remove.
3. Disable the associated Location Group in FCE to prevent item submission from the removed task.

Important: Removing a task does not delete its indexed data in Elasticsearch.

2.4 Creating a New Diskover Task After Removal

Q: Can I reuse an old task name?
No. Always use a new, unique name. Reusing old names may cause data from the old index to merge with new data.

Tip: Append a version or timestamp to ensure uniqueness (e.g., -v3, -2025-08).

2.5 Additional Best Practices

• Do not reuse task names—to avoid data blending between old and new runs.
• Plan task names carefully, as they appear in FCE with the fce- prefix.
• Understand file exclusions: files in directories like .Snapshot, files named Thumbs.db, .DS_store, desktop.ini, and 0-byte files are excluded.
• Update exclusion rules: edit home/diskover/.config/diskover/config.yaml (contact RecordPoint for assistance).

2.6 Changes from FCEv2

• Hash values are calculated only when files are “cracked” (content extraction enabled).
• The field indexing_date_fce is not present in v3.

3. Runtime Troubleshooting

3.1 Checking Service Status

Run the following command to verify that all connector services are active:

systemctl -a | grep -E 'IndexBuilder|rabbitmq|Connector|recordpoint-connector|elasticsearch|diskoverd'

If any services show as inactive, check their logs:

journalctl -u Connector.service

3.2 Checking RabbitMQ Queues

RabbitMQ is the primary messaging service between components.

1. Open http://localhost:15672/ on the FCEv3 VM.
2. Login:
   User: diskover
   Password: darkdata
3. Go to Queues and Streams and confirm queues exist and messages flow correctly.

3.3 Restarting Services

Restart Connector Services:

sudo systemctl restart recordpoint-connector-webapp.service
sudo systemctl restart Connector.service
sudo systemctl restart IndexBuilder.service

Restart Related Services:

sudo systemctl restart elasticsearch
sudo systemctl restart rabbitmq-server.service
sudo systemctl restart diskoverd.service

Alternative (all FCEv3 services):

bash /opt/recordpoint-connector-IndexBuilder/Deployment/fcectl.sh --restart

3.4 Reviewing Serilog Entries

Use the Elasticvue Chrome plugin to inspect serialog indexes. These logs often reveal runtime exceptions not captured in standard logs.

4. Miscellaneous Issues

4.1 Diskover Login Issues

Symptom: The login page reloads without an error after correct credentials.
Cause: This typically follows an nginx update.
Solution:

sudo chown -R nginx.nginx /var/opt/remi/php81/lib/php/session
sudo chmod -R 777 /var/opt/remi/php81/lib/php/session

4.2 Adjusting Log Levels

Default Log Level: Warning

Available Levels:

• Verbose – All traces, debugging, and system logs
• Debug – Debug-level and standard logs
• Information – Standard, warning, and error logs
• Warning – Warnings and errors only
• Error – Errors and exceptions
• Fatal – Critical system exceptions only

To Change Log Levels:

1. Open the appropriate appsettings.json file:
   • Connector: /opt/recordpoint-connector/appsettings.json
   • IndexBuilder: /opt/recordpoint-connector-IndexBuilder/appsettings.json
   • WebApp: /opt/recordpoint-connector-webapp/appsettings.json
2. Add or update the following block (ensure valid JSON, no trailing commas):

{
  "LoggerSettings": {
    "LogLevel": "Information"
  }
}

3. Restart FCEv3 services:

bash /opt/recordpoint-connector-IndexBuilder/Deployment/fcectl.sh --restart

4.3 Log File Locations

5. Elasticsearch Memory Management

5.1 Understanding Memory Usage

By default, Elasticsearch consumes up to 50% of system RAM, which can starve other processes.

5.2 Tuning for VMs with ≥32 GB RAM

Symptoms of memory starvation: Elasticsearch crashes and Out of Memory messages in dmesg.

Example:

[25006.929740] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/elasticsearch.service,task=java,pid=11083,uid=985
[25006.929843] Out of memory: Killed process 11083 (java) total-vm:24345380kB, anon-rss:17521756kB, file-rss:386204kB, shmem-rss:0kB, UID:985 pgtables:39520kB oom_score_adj:0

Recommended Actions:

1) Set Heap Size to 12 GB

echo -e "-Xms12g\n-Xmx12g" | sudo tee /etc/elasticsearch/jvm.options.d/heap.options > /dev/null
sudo systemctl restart elasticsearch

2) Add 6 GB of Swap Space

sudo fallocate -l 6G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon -p 10 /swapfile
echo '/swapfile none swap sw,pri=10 0 0' | sudo tee -a /etc/fstab

3) Optimize Swappiness

echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf > /dev/null
sudo sysctl --system

This configuration reduces memory pressure and prevents Elasticsearch from consuming excessive resources.