LabPP_Solaris Security Best Practices and Hardening Checklist

LabPP_Solaris Troubleshooting: Common Issues and Fixes

1. Boot failures or kernel panics

• Symptom: System stalls during boot, shows panic messages, or drops to single-user mode.
• Likely causes: Corrupted kernel or initramfs, incompatible kernel modules, disk corruption, recent configuration changes.
• Fixes:
  1. Boot from rescue media and check filesystem integrity (fsck).
  2. Restore a known-good kernel or initramfs from backup; remove recently added/third-party kernel modules.
  3. Review /var/adm/messages and dmesg for exact error strings to identify failing drivers.
  4. If hardware-related, run vendor diagnostics on memory and disks.

2. Network interface not coming up

• Symptom: No network on interface after reboot; ifconfig/ip shows interface down or missing.
• Likely causes: Misconfigured network scripts, wrong interface naming, driver/module not loaded, DHCP failure.
• Fixes:
  1. Check interface config files (e.g., /etc/hostname.or NetworkManager settings) and ensure correct persistent name.
  2. Manually bring interface up: ip link set dev eth0 up and obtain IP: dhclient eth0 or ip addr add.
  3. Confirm driver loaded: lsmod / modinfo ; load with modprobe.
  4. Inspect logs: tail -n 200 /var/log/syslog or journalctl for DHCP/NetworkManager errors.

3. Package installation or dependency failures

• Symptom: Package manager errors, unmet dependencies, failed installs.
• Likely causes: Repository misconfiguration, corrupted package cache, incompatible package versions.
• Fixes:
  1. Update repository metadata and clean cache: pkg update / pkg refresh –full or equivalent.
  2. Rebuild package database if available.
  3. Pin or explicitly install required dependency versions; use pkg install –reinstall .
  4. Check repository URLs and GPG keys; re-add or refresh keys if signature errors occur.

4. High CPU or memory usage by services

• Symptom: System slow, high load averages, swapping.
• Likely causes: Memory leaks, runaway processes, misconfigured service limits.
• Fixes:
  1. Identify culprits: top, htop, ps aux –sort=-%mem.
  2. Restart or gracefully reload misbehaving services; check their logs for errors.
  3. Tune service limits (ulimits, systemd service resource limits) or add swap if appropriate.
  4. Apply patches or update software if memory leaks are known bugs.

5. Storage full or unexpected disk usage

• Symptom: “No space left” errors; important services fail to write.
• Likely causes: Log growth, orphaned files, snapshots, temporary files.
• Fixes:
  1. Find large files: du -sh /* and find / -xdev -type f -size +100M.
  2. Rotate or compress logs; clear tmp directories.
  3. Check for snapshots (ZFS/Btrfs) consuming space and prune old ones.
  4. Expand filesystem or add storage if consumption is legitimate.

6. Service fails to start (systemd or init)

• Symptom: System reports service start failure, exit codes, or repeated restarts.
• Likely causes: Misconfiguration, missing dependencies, permission issues, port conflicts.
• Fixes:
  1. Inspect service status and logs: systemctl status and journalctl -u .
  2. Validate config files with built-in checkers (e.g., nginx -t).
  3. Check file permissions, SELinux/AppArmor denials, and socket/port availability.
  4. Run the service manually to surface runtime errors.

7. Authentication and access problems

• Symptom: Users cannot authenticate via SSH, LDAP, or local accounts.
• Likely causes: Incorrect PAM/SSSD configuration, expired keys, clock skew, network reachability to auth servers.
• Fixes:
  1. Verify PAM and SSSD configuration files and restart related services.
  2. Check SSH logs (/var/log/auth.log or journalctl) for authentication errors.
  3. Confirm system clock sync (NTP) and LDAP/AD server reachability.
  4. Test locally with passwd and su to isolate remote vs local issues.

8. Time sync drift

• Symptom: System clock drifting, causing cert or authentication failures.
• Likely causes: NTP/chrony service stopped, wrong timezone, hardware clock issues.
• Fixes:
  1. Ensure chrony/ntpd is running and sync status is healthy: chronyc sources or ntpq -p.
  2. Set timezone correctly and sync hardware clock: timedatectl set-timezone and hwclock –systohc.
  3. Check for virtualization host time issues.

9. Security alerts or unusual activity

• Symptom: Unexpected outbound connections, unknown user accounts, modified binaries.
• Likely causes: Compromise, misconfigured services, exposed management interfaces.
• Fixes:
  1. Isolate affected systems from network and preserve logs for forensics.
  2. Inspect running processes, network connections (ss -tunap), and recent auth logs.
  3. Run integrity checks (tripwire/aide) and compare binaries to known-good versions.
  4. Rotate credentials, update packages, and apply security patches; consider full rebuild if compromised.

10. I/O latency or disk errors

• Symptom: Slow disk I/O, I/O errors in logs, SMART warnings.
• Likely causes: Failing disk, misconfigured RAID, heavy I/O workload.
• Fixes:
  1. Check SMART data: smartctl -a /dev/sdX.
  2. Review kernel logs for I/O errors and identify failing device.
  3. Rebalance or replace failing disks; rebuild RAID arrays as needed.
  4. Tune filesystem mount options and I/O scheduler for workload.

Troubleshooting workflow (quick checklist)

1. Reproduce and capture exact error messages.
2. Check logs: system, service-specific, and kernel messages.
3. Isolate changes: recent updates, config edits, hardware swaps.
4. Test fixes in staging if possible, apply to production during maintenance windows.
5. Document root cause and remediation; add monitoring/alerts to detect recurrence.

If you want, I can convert this into a printable checklist or a step-by-step runbook for a specific LabPP_Solaris version — tell me the version and I’ll generate it.