Delete some Arabic letters in the file editor

Technical Issue Report: Arabic Characters Corruption in CyberPanel File Manager

Issue Description

When opening files containing Arabic text via the CyberPanel File Manager, some characters are missing or corrupted, while the same file displays correctly when accessed via SSH.

Symptoms

  1. Correct Output (via SSH):

    cat /home/example.com/public_html/text.txt
    

    Output:
    "كيف الحال يا من للحب مجال والسلام ختام" :white_check_mark:

  2. Corrupted Output (via File Manager):

    {status: 1, fileContents: "كيف الحال يا من للحب مجال والسلام خام", cont1: 37}
    

    Missing character: "ت" :cross_mark:

Debugging Findings

  1. The outputExecutioner() function retrieves file contents but loses some Arabic characters when processing the output.
  2. len() returns an incorrect count, confirming data loss.
  3. SSH works fine, meaning the issue is not in file storage but in how CyberPanel processes the output.

Attempted Fixes (Unsuccessful)

  1. Added UTF-8 encoding explicitly:

    p = subprocess.Popen(..., encoding='utf-8', errors='strict', env=env_vars)
    

    No improvement.

  2. Forced UTF-8 locale:

    env_vars['LANG'] = 'en_US.UTF-8'
    env_vars['LC_ALL'] = 'en_US.UTF-8'
    

    Still losing characters.


Request for Help

Since manual fixes haven’t resolved the issue, I’m seeking assistance from the CyberPanel community to:

Possible Investigation Paths

  1. Is there an encoding mismatch between:

    • The system locale (locale command)
    • Python’s default encoding (sys.getdefaultencoding())
    • The subprocess environment?
  2. Does the issue occur before or after outputExecutioner()?

    • If I log the raw bytes before decoding, are they correct?
  3. Is there a middleware (like gunicorn, nginx, or uWSGI) modifying the response?

  4. Does this happen with other Unicode characters (e.g., emojis, Chinese)?

Questions for the Community

  • Has anyone faced similar encoding issues in CyberPanel?
  • Are there known fixes for Arabic/Persian/Urdu text handling?
  • Should we modify the subprocess call or change how the response is sent to the frontend?

How to Reproduce

  1. Create a file with Arabic text.
  2. Open it via File Manager → Characters are missing.
  3. Check via SSH → Text is correct.

Conclusion

The issue appears to be in how CyberPanel processes command output, not in the file itself. Since my attempts to enforce UTF-8 failed, I’m looking for insights from developers who may have encountered this before.

Any suggestions or debugging tips would be greatly appreciated!

Relevant Code

The current outputExecutioner() implementation:

def outputExecutioner(command, user=None, shell=None, dir=None, retRequired=None):
    env_vars = os.environ.copy()
    env_vars['LANG'] = 'en_US.UTF-8'
    env_vars['LC_ALL'] = 'en_US.UTF-8'
    try:
        # ... (subprocess call with UTF-8 settings)
        output = p.communicate()[0].decode("utf-8")  # Still loses Arabic chars
        return output
    except Exception as e:
        logging.error(f"Encoding error: {e}")
        return ""

Operating System Information
CyberPanel 2.4.2 - The problem existed even before the update.
AlmaLinux 8

Looking forward to your ideas! :rocket:

Thank you for reporting this issue. The problem is related to UTF-8 encoding not being properly handled when reading files through the File Manager.

The issue occurs because the subprocess commands used to read files weren’t explicitly using UTF-8 encoding, which caused Arabic characters like “ت” to be lost or corrupted.

Fix implemented:

  1. Modified the file reading process to explicitly use UTF-8 locale (LANG=C.UTF-8 LC_ALL=C.UTF-8)
  2. Ensured the subprocess environment is set to handle UTF-8 properly
  3. Updated JSON responses to preserve Unicode characters

Changes made:

  • /filemanager/filemanager.py - Added UTF-8 locale to file reading commands
  • /plogical/processUtilities.py - Set UTF-8 environment for all subprocess calls
  • JSON responses now use ensure_ascii=False to preserve Unicode

I’ve fixed and pushed the code. Your Arabic text “كيف الحال يا من للحب مجال والسلام ختام” should display correctly with all characters intact after the update.

1 Like

Does this mean the problem is now solved and I should update to solve the problem?

yes please.

1 Like

I tried it now after the update, but the problem hasn’t been resolved.

Can you try the same text or any other Arabic text and see if it’s deleting any characters on your end?

how did you update?

Using

Upgrading CyberPanel

The following command can be used to upgrade to the latest version.

sh <(curl https://raw.githubusercontent.com/usmannasir/cyberpanel/stable/preUpgrade.sh || wget -O - https://raw.githubusercontent.com/usmannasir/cyberpanel/stable/preUpgrade.sh)

I checked the files and found that there are indeed updates.

I;ve checked on my end it displayed arabic correctly.

1 Like

Have you tried the same Arabic text?
ؤSo where could the problem be? Can you help me with that?
I found the same problem in another post.

i will check again

I’ve fixed and tested with text above.

1 Like

This topic was automatically closed 3 hours after the last reply. New replies are no longer allowed.