Delete some Arabic letters in the file editor

[nassar]

Technical Issue Report: Arabic Characters Corruption in CyberPanel File Manager

Issue Description

When opening files containing Arabic text via the CyberPanel File Manager, some characters are missing or corrupted, while the same file displays correctly when accessed via SSH.

Symptoms

1. Correct Output (via SSH):

   cat /home/example.com/public_html/text.txt
   

   Output:
   "كيف الحال يا من للحب مجال والسلام ختام"

2. Corrupted Output (via File Manager):

   {status: 1, fileContents: "كيف الحال يا من للحب مجال والسلام خام", cont1: 37}
   

   Missing character: "ت"

Debugging Findings

1. The outputExecutioner() function retrieves file contents but loses some Arabic characters when processing the output.
2. len() returns an incorrect count, confirming data loss.
3. SSH works fine, meaning the issue is not in file storage but in how CyberPanel processes the output.

Attempted Fixes (Unsuccessful)

1. Added UTF-8 encoding explicitly:

   p = subprocess.Popen(..., encoding='utf-8', errors='strict', env=env_vars)
   

   → No improvement.

2. Forced UTF-8 locale:

   env_vars['LANG'] = 'en_US.UTF-8'
   env_vars['LC_ALL'] = 'en_US.UTF-8'
   

   → Still losing characters.

---

Request for Help

Since manual fixes haven’t resolved the issue, I’m seeking assistance from the CyberPanel community to:

Possible Investigation Paths

1. Is there an encoding mismatch between:

   • The system locale (locale command)
   • Python’s default encoding (sys.getdefaultencoding())
   • The subprocess environment?

2. Does the issue occur before or after outputExecutioner()?

   • If I log the raw bytes before decoding, are they correct?

3. Is there a middleware (like gunicorn, nginx, or uWSGI) modifying the response?

4. Does this happen with other Unicode characters (e.g., emojis, Chinese)?

Questions for the Community

• Has anyone faced similar encoding issues in CyberPanel?
• Are there known fixes for Arabic/Persian/Urdu text handling?
• Should we modify the subprocess call or change how the response is sent to the frontend?

How to Reproduce

1. Create a file with Arabic text.
2. Open it via File Manager → Characters are missing.
3. Check via SSH → Text is correct.

---

Conclusion

The issue appears to be in how CyberPanel processes command output, not in the file itself. Since my attempts to enforce UTF-8 failed, I’m looking for insights from developers who may have encountered this before.

Any suggestions or debugging tips would be greatly appreciated!

Relevant Code

The current outputExecutioner() implementation:

def outputExecutioner(command, user=None, shell=None, dir=None, retRequired=None):
    env_vars = os.environ.copy()
    env_vars['LANG'] = 'en_US.UTF-8'
    env_vars['LC_ALL'] = 'en_US.UTF-8'
    try:
        # ... (subprocess call with UTF-8 settings)
        output = p.communicate()[0].decode("utf-8")  # Still loses Arabic chars
        return output
    except Exception as e:
        logging.error(f"Encoding error: {e}")
        return ""

Operating System Information
CyberPanel 2.4.2 - The problem existed even before the update.
AlmaLinux 8

Looking forward to your ideas!

[usmannasir]

Thank you for reporting this issue. The problem is related to UTF-8 encoding not being properly handled when reading files through the File Manager.

The issue occurs because the subprocess commands used to read files weren’t explicitly using UTF-8 encoding, which caused Arabic characters like “ت” to be lost or corrupted.

Fix implemented:

1. Modified the file reading process to explicitly use UTF-8 locale (LANG=C.UTF-8 LC_ALL=C.UTF-8)
2. Ensured the subprocess environment is set to handle UTF-8 properly
3. Updated JSON responses to preserve Unicode characters

Changes made:

• /filemanager/filemanager.py - Added UTF-8 locale to file reading commands
• /plogical/processUtilities.py - Set UTF-8 environment for all subprocess calls
• JSON responses now use ensure_ascii=False to preserve Unicode

I’ve fixed and pushed the code. Your Arabic text “كيف الحال يا من للحب مجال والسلام ختام” should display correctly with all characters intact after the update.

[nassar]

@usmannasir

intact after the update.

Does this mean the problem is now solved and I should update to solve the problem?

[usmannasir]

yes please.

[nassar]

I tried it now after the update, but the problem hasn’t been resolved.

Can you try the same text or any other Arabic text and see if it’s deleting any characters on your end?

[usmannasir]

how did you update?

[nassar]

Using

Upgrading CyberPanel

The following command can be used to upgrade to the latest version.

sh <(curl https://raw.githubusercontent.com/usmannasir/cyberpanel/stable/preUpgrade.sh || wget -O - https://raw.githubusercontent.com/usmannasir/cyberpanel/stable/preUpgrade.sh)

I checked the files and found that there are indeed updates.

[usmannasir]

I;ve checked on my end it displayed arabic correctly.

[nassar]

Have you tried the same Arabic text?
ؤSo where could the problem be? Can you help me with that?
I found the same problem in another post.

Arabic Language - Missing Letters when save work

What is happening as the following : I have only one language, which Is Arabic Language we are facing a problem that when we save our works on program file, some letters are missing from some words. We tried to fix it manually and put the missing letter again, but when we make(SAVE), the same problem happening again and again in another areas and words and another letters missing again. We did that manually many many times, but unfortunately the same issue showing.

[usmannasir]

i will check again

[usmannasir]

@nassar

كيف الحال يا من للحب مجال والسلام ختام

I’ve fixed and tested with text above.